Diagnosing Network Bottlenecks: One-sided Message Contention

Two trends suggest that one-sided message network contention is poised to become a cause of concern for scientific application developers. First, there is an increased interest in one-sided messages motivated by Global Address Space (GAS) programming models such as Unified Parallel C (UPC) [1], Co-Array Fortran (CAF) [2], [3], Global Arrays [4], and Chapel [5]. The GAS programming model provides a global shared address space abstraction along with ‘memory operations’ — such as load, store, and atomic increment — for distributed data structures. GAS programming models use one-sided messages for performance: such messages are readily implemented atop the Remote Direct Memory Access (RDMA) hardware of modern interconnects [6], [7]. However, RDMA operations must be used carefully because network interconnect latency is 1–2 orders of magnitude larger than a typical local memory access. Even worse, network contention can increase that latency by integer factors. Second, there is a growing ratio of hardware threads to network injection bandwidth [8]. When more than one hardware thread initiates a message at the same time, the messages may contend for network resources.